Papers by Hao Nan Sheng
AROMA: Autonomous Rank-one Matrix Adaptation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Low-rank adaptation (LoRA) and adaptive low-rank adaption (AdaLoRa) are effective for large language models but are expensive as model sizes escalate into hundreds of billions of parameters. |
| Approach: | They propose a framework that automatically builds up rank-one components with very few trainable parameters that gradually diminish to zero. |
| Outcome: | The proposed framework significantly reduces parameters compared to LoRA and AdaLoRA while maintaining subspace independence. |